Quantifying the robustness of causal inferences: Sensitivity analysis for pragmatic social science.

Social scientists seeking to inform policy or public action must carefully consider how to identify effects and express inferences because actions based on invalid inferences may not yield the intended results. Recognizing the complexities and uncertainties of social science, we seek to inform inevitable debates about causal inferences by quantifying the conditions necessary to change an inference. Specifically, we review existing sensitivity analyses within the omitted variables and potential outcomes frameworks. We then present the Impact Threshold for a Confounding Variable (ITCV) based on omitted variables in the linear model and the Robustness of Inference to Replacement (RIR) based on the potential outcomes framework. We extend each approach to include benchmarks and to fully account for sampling variability represented by standard errors as well as bias. We exhort social scientists wishing to inform policy and practice to quantify the robustness of their inferences after utilizing the best available data and methods to draw an initial causal inference.

Hypothetical case replacement can be used to quantify the robustness of trial results.

OBJECTIVES: We apply a general case replacement framework for quantifying the robustness of causal inferences to characterize the uncertainty of findings from clinical trials. STUDY DESIGN AND SETTING: We express the robustness of inferences as the amount of data that must be replaced to change the conclusion and relate this to the fragility of trial results used for dichotomous outcomes. We illustrate our approach in the context of an RCT of hydroxychloroquine on pneumonia in COVID-19 patients and a cumulative meta-analysis of the effect of antihypertensive treatments on stroke. RESULTS: We developed the Robustness of an Inference to Replacement (RIR), which quantifies how many treatment cases with positive outcomes would have to be replaced with hypothetical patients who did not receive a treatment to change an inference. The RIR addresses known limitations of the Fragility Index by accounting for the observed rates of outcomes. It can be used for varying thresholds for inference, including clinical importance. CONCLUSION: Because the RIR expresses uncertainty in terms of patient experiences, it is more relatable to stakeholders than P-values alone. It helps identify when results are statistically significant, but conclusions are not robust, while considering the rareness of events in the underlying data.

The Push and Pull of School Performance: Evidence from Student Mobility in New Orleans.

We investigate student mobility in a choice-based system that has gone to scale, New Orleans, to gain insight into an underlying improvement mechanism of choice-based reform and its potential equity-related consequences. In contrast to typical analyses of mobility, this study distinguishes incumbent school characteristics that can cause students to search for a new school ("push" factors) from those features that can draw families to a new school ("pull" factors). We find evidence consistent with school performance playing both push and pull roles. However, for low-achieving students, the push of low performance at incumbent schools is stronger than the pull of high performance at potential destinations, implying that low-achieving students are more successful in exiting low-performing schools than they are in finding higher-performing schools to attend.